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Models in the Design Phase 


Design Phase 


> Between the requirements phase and the implementation phase “The last you 
start the first you finish” 


> Produce models in order to clarify requirements and to better formalize them 
> Models can be the source of test set derivation strategies 


Various modeling notations for behavioral specification of a software system have 
been proposed. Which to use depends on the system you are developing, and the 
aspects you would like to highlight: 


@ Finite State Machines 

@ Petri Nets 

@ Statecharts 

@ Message sequence charts 
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Finite State Machines 


A finite state machine is a six-tuple <% , 4, 2, qo, 6, C> where: 
> X: finite set of input symbols 
> Y: finite set of output symbols 
> 2: finite set of states 
> Go E€ 2: initial state 
> ô: transition function (2 x 2 — 2) 
©: output function (2 x X — Y) 


Many possible extensions: 
@ Transition and output functions can consider strings 
@ Definiton of the set of accepting states F C 2 
@ Non determinism 


o = = = = 
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EEN 
Properties of FSM 


Useful properties/concepts for test generation 


> Completely specified (input enabled) 

o GE Las d) ds 2.ô(qi, a) = qj 

> Strongly connected 

o V(qi,q)) € 2x BAs € X*.0* (qi, $) = qj 

> V-equivalence (distinguishable) 

ə Let M, and Mo two FSMs. Let ¥ denote a set of non-empty string 
on the input alphabet 2°, and q; € 2; and qj € 22. qi and qj are 
considered / — equivalent if (ag, S) = G2(qj, S). If qi and q; are 
y — equivalent given any set / C @ than they are said to be 


equivalent (q; = qj). If states are not equivalent they are said to be 
distinguishable. 
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EE EE EE 
Properties of FSM....cntd 


Useful properties/concepts for test generation...cntd 

> Machine equivalence 

ə M and M: are said to be equivalent if Yqi € 21.3qj € 22.qi = qj 

and viceversa. 

> k-equivalence 
ə Let Mı and M. two FSMs and q; € 2; and qj € 2; and k € N. 
for Y ={sexXt||s|<k} 

> Minimal machine 


qi and qj are said to be .% — equivalent if they are / — equivalent 


e an FSM is considered minimal if the number of its states is less 
than or equal to any other equivalent FSM 
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Conformance Testing 


Conformance Testing 
Relates to testing of communication protocols. It aims at assessing 
that an implementation of a protocol conform to its specification. 
Protocols generally specify: 

> Control rules (FSM) 

> Data rules 
Developed techniques are equally applicable when the specification is 
refined into an FSM 
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The Testing Problem 


FSM and Testing 


> Reset inputs (2 = X U {Re}, and Y = Y U {null}) 
> Testing based on requirements checks if the implementation 
conforms to the machine on a given requirement. 


> The testing problem is reconducted to an equivalence 
(nevertheless finite experiments). Is the SUT (IUT) equivalent to 
the machine defined during design? 
> Fault model for FSM — given a fault model the challenge is to 
generate a test set 7 from a design My where any fault in M; of 
the type in the fault model is guaranteed to be revealed when 
tested against T 
ə Operation error (refers to issues with @) 
ə Transfer error (refers to issues with ô) 
ə Extra-state error (refers to issues with 2 and ô) 


ə Missing-state error (refers to issues with 2 and ô) 
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Mutation of FSMs 


A mutant of an FMS My is an FSM obtained by introducing one one or 
more errors one or more times. 

> Equivalent mutants: mutants that could not be distinguishable 
from the originating machine 


M Mut, Mut 2 Mut 4 
Co ED, ça 
©) © ©) io 
Ss 8 & 
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The Testing Problem 


Techniques to measure the goodness of a test set in relation to the 
number of errors that it reveals in a given implementation Mj. 


> Nt: total number of first order mutants of the machine M used for 
generating tests. 


> Ne: Number of mutants that are equivalent to M 
> N;: Number of mutants that are distinguished by test set T 
generated using some test generation method. 
> Ni: Number of mutants that are not distinguished by T 
The fault coverage of a test suite 7 with respect to a design M is 
denoted by FC(T,M) and computed as follows: 


FC(T, M) = Number of mutants not distinguished by T / 
Number of mutants that are not equivalent to M 
— (N: = Ne = N;)/(N: = Ne) J 
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Characterization Set 


Let M =< X,Y, 2,q1,6,@> an FSM that is minimal and complete. A 
characterization set for M, denoted as Y, is a finite set of input 


sequences that distinguish the behaviour of any pair of states in M. 
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K-equivalence partitions 


The notion of 4 — equivalence leads to the notion of 
A — equivalence partitions. 


Given an FSM a # — equivalence partition of 2, denoted by #, is a 
collection of n finite sets of states denoted as `k,, “ig, Xx, such 
that: 


> Ui EK, = 2 
> States in Ek for1isjsnare.4# — equivalent 


> if qı © Xk, and qm E Xx, for i # j, then qı and gm must be 
X — distinguishable 
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K-equivalence partitions 


The notion of .4# — equivalence leads to the notion of 
A — equivalence partitions. 


Given an FSM a # — equivalence partition of 2, denoted by #, is a 
collection of n finite sets of states denoted as X4,, 2 ,,..., Xx, such 
that: 
> Uit.n = 2 
> States in Ek for1isjsnare 4 — equivalent 
> if qı © Xk, ANd qm E Xx, for i # j, then qı and gm must be 
A — distinguishable 


neee nnee 
X — equivalence partitions can be derived using an iterative EE] 


for increasing number of #” 
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How to derive # from K-equivalence partitions 


@ Let M an FSM for which P = {P}, Po,..., Pn} is the set of k-equivalence partition. 
W =0 
@ Repeat the steps (a) through (d) given below for each pair of states (q;, qj), i # j, 
inM 
(a) Find r (1 < r < n such that the states in pair (qi, qj) belong to the same 
group in Pr but not in P,,,. If such an r is found then move to step (b) 
otherwise we find an n € 2 such that (qi, n) # O(q,7), set 
W = WU {n} and continue with the next available pair of states. The 
length of the minimal distinguishing sequence for (qi, qj) is r+ 1. 
(b) Initialize z = e. Let p; = q; and p2 = q; be the current pair of states. 
Execute steps (i) through (iii) given below for m = r,r — 1,...,1 
(i) Find an input symbol 7 in Pm such that (pi, n) 4 “(Po,n). In case 
there is more than one symbol that satisfy the condition in this step, 
then select one arbitrarily. 
(ii) set z = zn 
(iii) set ps = 6(p1,7) and pe = Ò(pe, n) 
(c) Find an n € 2 such that @(p1,7) # C(pe‚n). Set z = zn 
(d) The distinguishing sequence for the pair (qj, qj) is the sequence z. Set 
W=W U{z} 
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EEN 
Example 


@ Termination of the YW — procedure guarantees the generation of 
distinguishing sequence for each pair. 
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Example 


@ Termination of the / — procedure guarantees the generation of 
distinguishing sequence for each pair. 


x 
2 
N 
at 
A 
ep) 
Ka 


B WWNNDN = = = =H 
aanw AR WN AO 
w 
er 
-00000000 =|Y 

O — mk mk mk O) 


(Sollee ale nem ERNS dIE] 5. Test Generation — Finite State Models CS@UNICAM 13/31 


| 
The W-Method 


The W-Method aims at deriving a test set to check the implementation 
(Implementation Under Test - IUT) of an FSM model 
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| 
The W-Method 


The W-Method aims at deriving a test set to check the implementation 
(Implementation Under Test - IUT) of an FSM model 


> Mis completely specified, minimal, connected, and deterministic 
> M starts in a fixed initial states 


> M can be reset to the initial state. A nu11 output is generated by 
the reset 


> Mand IUT have the same input alphabet 


[=] 3 = = = 
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W-Method steps 


Given an FSM M =< X,Y, 2, qo,ò, C > the W-method consists of 
the following steps: 


@ Estimate the maximum number of states in the correct design 
@ Construct the characterization set # for the given machine .@ 


@ Construct the testing tree for .# and determine the transition 
cover set # 


@ Construct set # 
@ 2. F is the desired test set 
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Computation of the transition cover set 


2? -transition cover set 


Let q; and qj, i # j be two states of .#. Y consists of sequences s - x 


s.t. ô(qo,s) = qi A ô(qi, X) = qj forse %* Axe X. The set can be 
constructed using the testing tree for M. 


(Software Engineering Il — Software Testing) 
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Computation of the transition cover set 


P - transition cover set 
Let q; and qj, i # j be two states of æ. Y consists of sequences s- x 
st. ô(qo, S) = qi A ô(qi, X) = qj for se 2* Ax € X. The set can be 

constructed using the testing tree for M. 


Testing tree 
The testing tree for an FSM .# can be constructed as follows: 
@ State qo is the root of the tree 
@ Suppose that the testing tree has been constructed till level k. The 
(k +1)" level is built as follows: 
ə Select a node n at level k. If n appears at any level from 1 to k — 1 
then nis a leaf node. Otherwise expand it by adding branch from 


node nto a new node m if #(n, x) = m for x € 2. This branch is 
labeled as x. 
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Constructing & 


Suppose number of states estimates to be m for the IUT, and n in the 
specification m > n. We compute # as: 


Z = (£2. W)U(B-W)U(B'-W) ULKI. WUA TN.) 
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Deriving a test set - #. 2 
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Deriving a test set- A. Z 


Try sequences: 
> baaaaaa 


> baaba 
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W -method fault detection rationale 


> A test case generated by the WY — method is of the form r-s 
where r € P and sE YW 
ə Why can we detect operation errors? 
ə Why can we detect transfer errors? 


2? = {e, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa} 
W = {a, aa, aaa, baaa} 


o a = = 2 Dar 


(Stor NEETER tale ale Late RIE KG NEIGES NN] 5. Test Generation — Finite State Models CS@UNICAM 19/31 


W -method fault detection rationale 


> A test case generated by the WY — method is of the form r - s 
where r € P and sE Y 
ə Why can we detect operation errors? 
ə Why can we detect transfer errors? 


2? = {e, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa} 
W = {a, aa, aaa, baaa} 
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W -method fault detection rationale 


> A test case generated by the WY — method is of the form r - s 
where r € P and sE Y 
ə Why can we detect operation errors? 
ə Why can we detect transfer errors? 


2? = {e, a, b, bb, ba, bab, baa, baab, baaa, baaab, baaaa} 
W = {a, aa, aaa, baaa} 


Dar 
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The partial YW — method (aka Wp — method) 


Wp — method 
Main characteristics: 

> |t considers minimal, complete and connected FSM 

> is inspired by the # — method it generates smaller test sets 


> uses a derivation phase split in two phases that make use of state 
identification sets % instead of characterization set YW 


> uses the state cover set (.7) to derive the test set. 


AAC 
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Identification Set and State Cover Set 
Identification Set 


and has the following properties: 


The Identification Set is associated to each state q € 2 of an FSM. 
An Identification set for state q; € 2, where |2| = n, is denoted by 7 


B HEN peri <is<n 


@ 3s1sjsnAsE AO, SLAG, s) 
@ No subset of #; satisfies property 2. 
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Identification Set and State Cover Set 


Identification Set 
The Identification Set is associated to each state q € 2 of an FSM. 


An Identification set for state q; € 2, where |2| = n, is denoted by 7 
and has the following properties: 


PB HCN peri<isn 
@ 3s1sjsnAsE AO, SLAG, s) 
@ No subset of ⁄ satisfies property 2. 


State Cover Set 

The state cover set is a nonempty set of sequences (S C #* st: 
> Va €E Zar € Sstòlgo,r) = qi 

From the definition it is evident that 7% C # 


o S = = 
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The WY p procedure (assuming m = n) 


The test set derived using the Wp — method is given by the union to 
two test sets 7, % calculated according to the following procedure: 


@ Compute sets Z, Z, W, and H 

O =F- W 

Q Let W ={H,%,...,Wn} 

@ Let R={h,h,...,m~} where R = P — F and rj € Ris st. 
ò(qo, rj) = qi 

@ %Z=ROW=VA,({G}-%) where #; € W is the state 
identification set for state q; 
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W p — method rationale 


@ Phase 1: test are of the form uv where uc Z and v €. Reach 
each state than check if it is distinguishable from another one 

@ Phase 2: test covers all the missing transitions and then check if 
the reached state is different from the one specified in the model 
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W p — method in practice 


W = fa, aa, aaa, baaa} 

2? = {e, a,b, bb, ba, bab, baa, baab, baaa, baaab, baaaa} 
S = {e,b, ba, baa, baaa} 

#1 = {baaa, aa, a}, Wo = {baaa, aa, a}, Wz = {aa, a} 
Wa = {aaa, a}, Ws = {aaa, a} 


Oo a LS = 
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W p — method in practice 


W = fa, aa, aaa, baaa} 

2? = {e, a,b, bb, ba, bab, baa, baab, baaa, baaab, baaaa} 
S = {e,b, ba, baa, baaa} 

#1 = {baaa, aa, a}, Wo = {baaa, aa, a}, Wz = {aa, a} 
Wa = {aaa, a}, Ws = {aaa, a} 
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W p — method in practice 


W = fa, aa, aaa, baaa} 

2? = {e, a,b, bb, ba, bab, baa, baab, baaa, baaab, baaaa} 
S = {e,b, ba, baa, baaa} 

#1 = {baaa, aa, a}, Wo = {baaa, aa, a}, Wz = {aa, a} 
Wa = {aaa, a}, Ws = {aaa, a} 
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Let’s consider the following FSM: 
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The WY p procedure (assuming m > n) 


Modify the derivation of the two sets as follows: 


> h=. ZE where 2 = Z[m—-n|-H 
> D=(R. Xim- n)8W 


e Let S=R. X[m- n] = {s|s=r-ust. reRAue Zim- nj} 


then P =S QY = User(s: Wi) where ô(qo, S) = 5(0(o, r), u) = qi 


(mi 
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UlO-Sequence Method 


UIO-Sequence 


A UIO sequence is a sequence of input and output pairs that 
distinguish a state of an FSM from the remaining states. 

UIO(S) = ty /04, /0%,..., inf On St. 

Vie ZNS F taj € [1 eae n]. (ds, hip... i_1), ij) £ O(ólt, hi... Ì1), ii) 


M is completely specified, minimal, connected, and deterministic 
M starts in a fixed initial states 


M can be reset to the initial state. A nu11 output is generated by 
the reset 


> Mand IUT have the same input alphabet 
> Mand IUT have the same number of states 


v 


v 
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Distinguishing Signatures 


Distinguishing Signature or Sequence (DS) 
Sequence of input/output labels that is unique to a state s 
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Distinguishing Signatures 


Distinguishing Signature or Sequence (DS) 
Sequence of input/output labels that is unique to a state s 


Minimal transfer sequence 


A minimal transfer sequence is a sequence of input/output that brings 
the machine from state j to state í along the shortest path P;(/) 


Given a state i a DS can be built using the identification set and 
minimal transfer sequences for each state j with j # i . In particular for 
an FSM M with k states a DS is given by the following concatenation: 
DS(qi) = W(qi, qi) - Pi(k) - W( Qi, q2)--- Pi(tk-1)W (qi, qk) 


fa = = = = 
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Test generation 


Let M = (2, X , Y, q, ô, 0) an FSM and & = {(qi, X,Y, 9) 19, qj € 
Q2NXERNVEYAGG,X) = G A (qi, x) = y} the set of edges of 
M 


@ Find the UIO for each state in M 


@ Find the shortest path from the initial state to each of the 
remaining states. 

@ For each edge e = (qj, X, y, qj) € £, build 
TE(e) = Pheaa(e)(1) - label(e) - UIO(tail(e)) 
where head(e) = qj, tail(e) = qj, label(e) = x/y 

@ Optionally a unique sequence can be derived using reset actions. 
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Assessment of automata theoretic strategies 


Control Flow based techniques are typically assessed according to 
different criteria: 


State coverage 


A test set T is considered adequate with respect to the state cover 
criterion for an FSM M if the execution of M agianst each element of T 
causes eash state in M to be visited at least once 


Transition coverage 


A test set T is considered adequate with respect to the branch, or 
transition, cover criterion for an FSM M if the execution of M against 
each element of 7 causes each transition in M to be taken at least 

once 
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Assessment of automata theoretic strategies 


Switch coverage (n-switch coverage) 


A test set T is considered adequate with respect to the 1-switch cover 
criterion for an FSM M if the execution of M against each element of T 
causes each pair of transition (tr, tro) in M to be taken at least once, 
where for some input substring ab € X*, in : qi = 5(qj, a) A 

tho : Ox = ô(qi, b) and qi, qj, qk are states of M 


Boundary-interior coverage 


A test set T is considered adequate with respect to the 
boundary-interior cover criterion for an FSM M if the execution of M 
against each element of 7 causes each loop body to be traversed zero 
times and at least once. Exiting the loop upon arrival covers the 
“boundary” condition and entering it and traversing the body at least 
once covers the “interior” condition. 
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